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ABSTRACT 



A study evaluated a new freshman writing course offered to 
students at the University of Minnesota who were underprepared for study at a 
major research university. In the fall of 1997 it was proposed that all 
freshmen be placed in for-credit, strengthened, basic writing courses rather 
than placing certain students in non-credit, pre-college-level remediation 
courses. The study then examined course completion rates, students' attitudes 
toward the courses, students' performance in future courses, and growth in 
student writing. At the end of the first year, 87 percent of students had 
successfully completed the basic writing course. Additionally, when students 
were asked to compare the basic writing course with similar high school 
courses, the results showed that they were completing the basic writing 
course because they felt challenged and not because the course was easy. A 
review of students' grades in other writing courses found that students 
taking the basic writing course were well prepared for future composition 
courses. And student papers rated by trained raters showed that students were 
performing important writing tasks at the expected level . The study concluded 
that the new writing program was achieving its goal of preparing all freshmen 
for basic writing at the college level. (CH) 
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Evaluating Freshman Composition: A Multimethod Approach 



Abstract 

This paper describes 4 studies that were used to evaluate the freshman writing 
courses offered to basic writers at a major research university. The evaluation was 
prompted by an institutional decision to eliminate post-admission writing placement and 
place all freshmen into college level writing courses. The studies investigated course 
completion rates, performance in subsequent courses, student opinion of course difficulty 
and analysis of student writing. The studies suggest that the courses are effective in 
achieving their goals. 




General College (GC) at the University of Minnesota is the first home for 800- 
900 new freshmen each year. The college's mission is to provide the first year of 
instruction for students who are underprepared for study at a major research university. 
During their year in GC, students have the opportunity to demonstrate their ability to 
perform in a challenging, yet supportive environment. 

All students in GC are required to take 2 quarters of freshman composition. Both 
courses are college level, credit-bearing courses (as determined by the State Higher 
Education Coordinating Board). The decision to place all GC freshmen directly into a 
college level writing curriculum regardless of admissions test scores and high school rank 
was a radical one, given the nationwide support for mandatory remediation programs 
(Grabel, 1988; Rounds & Anderson, 1985). Evaluations of the curriculum have 
demonstrated, however, that the vast majority of students, even those considered very 
underprepared students, can successfully complete the curriculum with college level 
writing skills. 

The history of the decision 

In the mid-1980s, GC faculty in the writing area became convinced that the 
college's writing program was not working. At that time entering students were given a 
paper-and-pencil reading test (the Descriptive Test of Language Skills) and a writing test 
(the Written English Proficiency Test). Based on the results of those tests, students were 
recommended for pre-college or college-level reading courses, and pre-college or 
college-level writing courses. At least 60% of entering freshmen were placed in pre- 
college-level courses. While placement was not mandatory, many students did follow the 
placement suggestions and enrollment in pre-college-level reading and writing courses 
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was high. The college's reading and writing courses were administered through learning 
assistance centers and involved combinations of self-paced and classroom instruction. 
Faculty were skeptical of the curriculum because the placement process had not been 
validated by research, the courses in the program did not appear to be of good quality, the 
rate for non-completion of the courses was high, and there was no evidence that students 
who completed the courses performed better in subsequent writing courses. 

Two possible solutions to the problem were discussed. The first solution, the one 
adopted by most institutions (and mandated by some state legislatures) was to strengthen 
the placement process and the curriculum. The most credible writing placement 
technologies at the time were based on holistic scoring of essays. This process was very 
expensive to implement and there was no guarantee that the validity of the placement 
process would be significantly improved. The faculty made efforts to strengthen the pre- 
college-level courses, and while completion rates improved, students were unhappy about 
the non-credit status of the courses, and the faculty remained unconvinced that they were 
necessary. The issue came to a head when the director of the learning assistance center, 
who was the main supporter of the placement process, left the university. The faculty 
had to decide whether to fix the existing system or revise it completely. 

The writing faculty proposed that the placement process be eliminated and that all 
freshmen be placed in a strengthened basic writing course. The rationale for the decision 
resided in emerging theories of writing process (cf. Bartholomae, 1986, and Rose, 1983). 
These theories suggested that immersing students in the writing process was a more 
effective way to develop writers. The course proposed would meet for 6 hours per week 
(for four credits) with 1 8 students per section in computer classrooms featuring 
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interactive software. Students who failed to complete the basic writing course would 
register for a non-credit, independent-study writing course that would be supervised by 
staff in the writing center. Students would receive a passing grade for basic writing upon 
completing the latter course. 

Two concerns emerged about the plan. The first concern was whether a basic 
writing course could be designed that would be rigorous enough to be considered college 
level and therefore worthy of credit. The second concern was whether underprepared 
freshmen could pass such a course. The first concern was addressed in a preliminary way 
when the state of Minnesota Higher Education Coordinating Board (HECB) endorsed the 
course as college-level.o Both questions have been addressed through research that has 
examined course completion rates, students' attitudes toward the courses, students' 
performance in future courses, and growth in student writing. 

Course completion 

The curriculum was first offered to students in the fall of 1987. Anecdotal 
evidence from writing teachers suggested that the curriculum was working, and at the end 
of the year, about 87% of students had successfully completed the basic writing courses. 
These completion rates have been sustained over subsequent years. For example, in 1995 
Wambach and delMas examined the grades of GC students registered in GC courses 
taught in day school between fall 1993 and spring 1994. Information on grades was 
obtained from the U of M Student Registration database. Grades of A,B,C,S, and D are 
considered passing. Grades of F, N, I, and W are considered not passing. About 87% of 
GC students passed the courses (see Table 1), which is higher than the average 
completion rate for all courses in the college (75.8%). 
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Table 1: Course completion rates for first and second quarter freshman composition 



Course 


Grade of 


Grade of 


Grade of 


Total 




A,B,C,S 


D 


F,N,I,W 


enrollment 


First Quarter 


575 


13 


85 


673 


Writing 


85.4% 


1.9% 


12.6% 




(GC 1421) 










Second Quarter 


522 


18 


74 


614 


Writing 


85% 


2.9% 


12.1% 




(GC 1422) 











The extended course evaluation 

Information on the extent to which students believed their writing courses to be 
challenging was gathered as part of a larger survey of students' beliefs about the 
curriculum. In the spring of 1995, students enrolled in General College were asked to 
compare their GC courses to those taken in high school (Wambach, Thatcher & Woods, 
1996 ). The survey asked students to compare how the GC courses they were currently 
enrolled in compared to high school courses and for their overall impressions of their GC 
courses. GC instructors distributed the surveys in their courses during the last week of 
the quarter as part of a routine course evaluation. Responses were received from 136 
students enrolled in GC 1422, second quarter writing. The respondents represent about a 
third of the students enrolled in the course that quarter. Since the surveys were 
completed anonymously as part of the course evaluation process, it was not possible to 
compare the characteristics of respondents and non-respondents. Participation in the 
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survey was affected by absenteeism from class and instructor willingness to participate. 
Thus the respondents represented a large, but not a random sample of GC students. 

Question 3 on the survey asked students to rate "How does the difficulty of the 
homework assignments in this course compare to the difficulty of the homework 
assignments in a similar high school course?" Students in 1422 reported that the course 
was much more difficult than high school writing courses (see Table 2). Students were 
also asked to compare the amount of work required to complete projects and papers in 
this course to the amount of work required to complete projects and papers in a similar 
high school course. Students in GC 1422 indicated that they were doing much more 
work in this course than they had in their equivalent high school courses. Results from 
the survey suggested that most GC students were challenged by their experiences in 
writing courses, suggesting that students completed the courses even though they were 
challenging and not because they were too easy. 

Table 2. Comparison of GC 1422 to high school writing courses 



Survey Question 


Much less than 
high school 


About the same 
as high school 


Much more than 
high school 


How difficult was the course? 


5 


38 


93 




3.7% 


27.9% 


68.4% 


How much work did you do in 


6 


27 


103 


this course? 


4.4% 


19.9% 


75.7% 
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Grades in other writing courses 

While data on course completion and student perceptions suggested that the 
writing courses were challenging students and effectively retaining them, questions 
remained about the extent to which the courses were effectively preparing students to 
write in subsequent courses. In order to address this issue, a study was conducted to 
determine whether or not General College composition courses prepared students for 
future composition courses at the University of Minnesota (delMas, 1994). The 
University of Minnesota registration database was used to identify all students who had 
taken advanced composition courses including COMP 1027, RHET 3562, or any 3000- 
level COMP course. The data set was then limited to those students who started at the 
University as of fall 1987 or later. Fall 1987 was selected because it represented a point 
in time when GC 1422 was being taught in a way similar to the approach used at the time 
of the study. Finally, not all 3000-level COMP courses were used in the study. Several 
courses had either no or only a few (1 to 5) students who started in the General College. 
These courses were eliminated from the data set. The final result was a data set with 
9,164 students, 553 of whom started in the General College and who had taken GC 1422, 
second quarter writing. 

Table 3 presents a picture of the grade distribution among the different advanced 
composition courses. The vast majority of students (60% or more) earned grades of B or 
higher. Several things are worth noting in Table 3. The most common advanced 
composition courses for GC students appear to be COMP 1027, COMP 3014, and RHET 
3562. GC students do well in these courses, with about 70% earning As or Bs in the two 
COMP courses, and 81% earning As or Bs in RHET 3562. Table 4 points out that while 
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GC students don't perform as well as non-GC students in their first advanced composition 
course, GC student performance is still quite high and comparable (75% As and Bs for 
GC compared to 88% As and Bs for non-GC students). 

delMas also looked at the relationship between GC 1422 and composition course 
grades, presented in Table 5. Here, grades in GC 1422 are crosstabulated with grades in 
the first composition course a student took after taking GC 1422. It is rare for a student 
to earn a D or F in a composition course, regardless of the student's grade in GC 1422. 
The likelihood of earning a grade of A does increase, however, as grade in GC 1422 
increases, as does the likelihood of earning a B or higher in the composition course. This 
is probably the strongest evidence that suggests a positive relationship between GC 1422 
and future composition course performance. 



Table 3: Distribution of grades in composition courses for students who started in 
General College. 



Composition 

Course 


Sample 

Size 


% A 


% B 


%C 


% D 


% F 


COMP 1027 


126 


16 


54 


26 


2 


2 


COMP 3011 


37 


22 


49 


27 


0 


3 


COMP 3012 


19 


16 


47 


37 


0 


0 


COMP 3013 


32 


19 


66 


16 


0 


0 


COMP 3014 


117 


16 


53 


29 


1 


1 


COMP 3015 


17 


12 


65 


18 


0 


6 


COMP 3022 


21 


10 


71 


19 


0 


0 


COMP 3027 


80 


16 


61 


23 


0 


0 


COMP 3031 


16 


31 


56 


6 


0 


0 


COMP 3033 


32 


34 


50 


16 


0 


0 


RHET 3562 


106 


25 


56 


15 


0 
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Table 4: Comparison of grades in first composition course (other than GC 1422) between 
students who started in General College and those who did not. 



Started 


Sample 












in GC 


Size 


% A 


%B 


%C 


% D 


% F 


YES 


553 


18 


57 


22 


1 


1 


NO 


8,611 


39 


49 


9 


1 


1 




1 



2 



10 



Table 5: Comparison of grades in GC 1422 with grades in first composition course after 
taking GC 1422. 







Grade in Composition Course 


GC 1422 
Grade 


Sample 

Size 


% A 


% B 


% C 


% D 


% F 


A 


81 


32 


54 


14 


0 


0 


A- 


81 


19 


64 


14 


0 


4 


B 


150 


15 


61 


23 


1 


1 


B-, C+ 


91 


7 


62 


29 


1 


2 


C, C-, D 


32 


13 


41 


44 


3 


0 



While this study suggested that students who successfully completed GC 1422 
were well prepared for future composition courses, it did not tell us if the GC 1422 
experience was responsible for students' future success. It could be that students who go 
on to advanced composition courses are already good writers before entering college, and 
that successful completion of 1422 is simply another measure of this prior preparation. 
Analysis of student writing study 

The most recent study of the college writing curriculum has been an examination 
of the development of students' writing skills (Adler-Kassner, Reynolds & delMas, 

1996). While prior research has suggested that students successfully complete the 
writing courses, find them challenging, and go on to successfully complete advanced 
writing courses, we did not have evidence that students' writing actually improved 
through their experience in the course. The purpose of this study was to test the 
hypothesis that student writing is affected by experience in GC writing courses. 



All students enrolled in GC 1421 and GC 1422 during winter quarter 1996 were 
asked to participate in the writing program study by allowing writing program staff to 
photocopy course papers for rating. For students enrolled in GC 1421, three papers were 



photocopied (papers one, two, and four); for students in GC 1422, two papers were 
photocopied (papers two and four). Only students whose sets of papers were complete 
for each class were included in the study. From the several hundred complete sets of 
papers, a sample of 56 sets was drawn from 1421 and 1 10 sets were selected from 1422 
for analyses. The sample sizes were determined by the budget allotted for payment to 
raters. Students of color were over-represented in the sample. In other respects, the 
sample formed a group that was statistically indistinguishable from those not included in 
terms of grade point average and demographic indicators. 

The papers were evaluated by trained raters during the summer of 1996. Raters 
were hired from among staff in Commanding English, an ESL-based program in General 
College. The raters were chosen because, while they were familiar with the theory and 
practice of basic writing and basic writers, they were not entirely familiar with the 
writing program's focus and assignments. Raters were trained in early June. The training 
period included rating exercises that were designed to increase the reliability of ratings. 
After their training, they rated approximately twenty papers daily for ten days. During 
the period they were rating papers, the study authors met with the raters every other day 
to address pertinent questions. Raters received papers with assignment sheets stapled to 
the front and were trained in finding relevant criteria from those sheets for their rating. 
Raters were asked to evaluate four aspects of each paper: 

1) length of paper (one question); 
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2) structure, development, and organization (four questions); 

3) use and documentation of sources (two questions); 

4) surface issues (one question). 

Each aspect was rated on a six-point scale running from 1 (very little or no 
evidence of the identified characteristic) to 6 (exceptional evidence of the identified 
characteristic). A rating of 3 was assumed to be the mid-point, or average, rating on the 
scale. A rating of 6 was assigned when the paper clearly went beyond the expectations 
stated on the assignment sheet; a rating of 1 was assigned when it clearly was far below 
those expectations. Papers were rated in random order and were not identified by course, 
section, instructor, or order of assignment. Any identifying information about the student 
writer was removed. 

Inter-rater agreement was between 79% and 86% for each area rated. The rating 
assigned for the analysis was the average of the two raters' scores. 

Results of the study suggested that most students wrote a paper that was the 
appropriate length for the assignment (neither too short nor too long), and that the 
number of students meeting this criterion did not change as the students moved through 
the curriculum (see Table 6). 

Table 6: Percentage of students who wrote papers of an appropriate length (for the 
assignment): 



Course 


Paper 1 


Paper 2 


Paper 4 


1421 


74% 


81% 


64% 


1422 




74% 


68% 
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One of the surprising findings of this study was that the average ratings for first 
papers in GC 1421 were at expectation. Given that this group of students scores lower 
than average university freshmen on standardized tests of verbal ability (their mean score 
on the ACT English test was 18.9), we expected to see lower ratings of initial writing 
samples. Ratings of structure and organization, use and documentation of sources, and 
surface features remained nearly constant in GC 1421 and GC 1422 (see Table 7). Both 
at the beginning and end of the courses, students average ratings were 3 or 4, which is at 
expectation. At the end of 1421, students were improved from 3.6 to 4 on structure and 
organization. Virtually no improvement was seen in ratings of use and documentation of 
sources or surface features. At the end of GC 1422, students average score on structure 
and organization improved very modestly from 3.6 to 3.8. Ratings of use and 
documentation of sources and surface features changed slightly in a positive direction. 

One explanation for the lack of observed change is that it takes a great deal of 
practice to improve one's writing. Ten weeks of course work is probably not enough to 
create significant improvements in this skill. Another explanation is that the writing tasks 
assigned in the courses tended to increase in difficulty and complexity throughout the 
quarter. In GC 1421, for example, the first paper is a narrative, autobiographical writing 
assignment while the fourth paper is a research report. 
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Table 7: Mean ratings for papers' structure and organization, use and documentation of 
sources and surface features in GC 1421and GC 1422. 



Criterion 


1421 


1422 




Paper 1 


Paper 2 


Paper 4 


Paper 2 


Paper 4 


Structure and 
Organization 












The paper clearly addresses 
the assigned question 


3.8 


4.0 


4.1 


3.8 


3.9 


Easy to infer from the first 
page the main point of this 
paper 


3.9 


4.0 


4.4 


3.9 


4.1 


The paper's progression 
proceeds from, furthers, and 
is logically connected to the 
main idea announced on the 
first page 


3.4 


3.7 


3.6 


3.4 


3.7 


The evidence used in this 
paper supports the thesis 
effectively 


3.5 


3.6 


3.8 


3.4 


3.6 


Use and Documentation of 
Sources 












The paper presents, 
analyzes and discusses 
sources effectively 
according to the instructions 
on the assignment sheet 


3.5 


3.1 


3.5 


3.1 


3.3 


The evidence in the paper is 
appropriately documented 
according to the instructions 
on the assignment sheet 




3.2 


3.3 


3.1 


3.2 


Surface Features 












The paper has an 
appropriately low incidence 
of composing and editing 
errors 


3.5 


3.5 


3.6 


3.4 


3.5 
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Conclusions 



The series of investigations described above have persuaded the faculty that the 
decision to place all GC freshmen in a college-level basic writing sequence was 
appropriate. Students complete the courses, find them challenging, are prepared for 
future courses and show evidence that they are performing important writing tasks at 
expectation. This information gives us confidence to continue to improve our writing 
program within it's existing framework. 
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